This invention relates to a computer-based, artificial neural network system for learning, recognizing, and generating temporal-spatial sequences. Use of an array of associative neural networks (ANNs) permits manipulation of complex sequences.
Investigations of neural processing in biological systems have provided information from which artificial neural networks are being developed. However, what evolution has generated, man finds difficult to emulate. Artificial networks are implemented through software and/or hardware to perform functions analogous to those performed by living organisms (for example, pattern recognition and classification), but there are many limitations precluding broad applications to problems of commercial interest.
Conversely, an objective of constructing and testing artificial neural networks, is to develop an understanding of biological neural networks. This information is particularly useful in appreciating and treating human neurological disorders. Many recent observations in the field of neurobiology suggest that biological neurons are much more complex than the kinds of model neurons used in artificial neural networks in the art. Some examples of the complexity are that the learning rule need not be Hebbian (Hebb, 1949), and that learning can occur locally and independently of whether the post-synaptic neuron fires. Furthermore, even to make one association, memory involves the interaction of changes in more than one spatially distinct compartment of the same neuron.
A component which needs to be factored into artificial networks to increase their applicability is the ability of a brain to continuously process temporal information, as the environment changes over time. In particular, the brain routinely and dynamically learns and recalls information. Therefore, integrating temporal adaptive processes analogous to those operative in a brain is a major goal in constructing useful artificial neural networks (ANNs). ANNs have been developed for static processing and for temporal processing. However, systems are not yet available which incorporate temporal processing at a level of complexity that is suitable for useful applications such as the classification of temporal signals. Temporal patterns represented by signals include those generated from time-varying spatial patterns.
Temporal processing includes one or more of the following functions: learning, recalling, classifying, generalizing, or generating time-dependent phenomena. Neural networks that have been developed to perform temporal processing may be divided into two categories: (i) those with time delays and (ii) those without time delays.
Time delays are implemented in a variety of systems, including physical and chemical systems, as well as artificial neural networks. Experimental evidence for neural signal propagation delays has been found in rat hippocampus. Further work is needed to elucidate whether time-delay is actually used in temporal processing in the brain. A theoretical hippocampal model with time-delays has been proposed by Zipser (1986).
Time-delays have been proposed to represent temporal sequences. For example, Fukushima (1973) presented a temporal processing system, in which a number of McCulloch-Pitts neurons are fully connected with Hebbian-type synapses. McCulloch-Pitts neurons (McCulloch and Pitts, 1943) are non-linear processing elements that have two states, i.e., firing and quiescent. Each neuron receives signals from its neighboring firing neurons, and the signals are transmitted through synaptic weights. The neuron then either fires if the total input exceeds a threshold, or remains quiescent. A Hebbian-type synapse is a synapse whose strength increases when the two neurons connected by the synapse fire together at a given instance during learning, and conversely, decreases when only one of the two neurons fires and the other remains quiescent. There are multiple synapses between any two neurons and different time-delays in these synapses.
Fukushima's system operates by associating a spatial pattern with a pattern present at a previous time. However, this formulation has only a limited ability to store sequences, i.e., it is rapidly saturated. Furthermore, this system requires many iterations for sequence retrieval and has great difficulty discriminating non-orthogonal patterns. Non-orthogonal patterns are those for which the mathematical relationship of the vector of binary signals is that their product is not zero, that is, they are not independent. This is in comparison with orthogonal vectors whose product is zero and are independent. Furthermore, images retrieved by this system are often obscured by noise. This noise is referred to as "spurious memories."
Time delays have been incorporated into Hopfield networks (Hopfield, 1982) to generate temporal-spatial sequences (Sompolinsky and Kanter, 1986; Kleinfeld, 1986; Tank and Hopfield, 1987). These systems also use Hebbian learning rules and have problems similar to those of Fukushima's system. The ANN discussed by Guyon et al. (1988) requires that all stored sequences are known analytically a priori. After synaptic connections are calculated, any additional sequences that need to be stored in the system require reconstruction of the entire synaptic connectivity.
Time delays have also been used together with back-propagation networks in processing temporal speech signals (Lippmann, 1989), although back propagation networks are known to have unacceptably long training times due to iterative learning procedures. Other iterative learning algorithms include that used by Unnikrishnan et al. (1991).
A number of ANNs have been reported to generate temporal sequences without time delays. Stochastic noise has been used to induce transitions between attractors in Hopfield networks (Buhmann, 1987). Other existing mechanisms are time-dependent (Peretto and Niez, 1985; Dehaene et al., 1987), asymmetric (Coolen and Ruijgrak, 1988; Nishimori et al., 1990), and diluted higher order synaptic interactions (Wang and Ross, 1990 a, b; 1991 a, b, 1992). But it is not yet straightforward to train these ANNs for practical applications, such as classifications of temporal signals. Limitations on systems are that single values, rather than arrays of data are output; only single neural networks have been used, limiting processing to orthogonal spatial images and data sets; and complex sequences encounter storage limits.
The present invention relates an artificial neural network system which overcomes these limitations by employing a time-delay signal processing method and an array of neural subnetworks. The system may be incorporated into a general neural network such as the DYSTAL (Dynamically Stable Associative Learning Network) associative neural network (Alkon et al., 1990) for the purpose of learning temporal associations. Unlike previously proposed temporal systems, the present invention relates a parallel array of neural subnetworks and a comparator layer to determine the overall output of the network. This design is novel and provides for several advantageous performance features.